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September  6,  2013 


Congressional  Committees 

Department  of  Defense’s  Waiver  of  Competitive  Prototyping  Requirement  for  the  VXX 
Presidential  Helicopter  Replacement  Program 

The  Weapon  Systems  Acquisition  Reform  Act  of  2009,  as  amended  (WSARA),  requires  the 
Secretary  of  Defense  to  modify  guidance  to  ensure  that  the  acquisition  strategy  for  each  major 
defense  acquisition  program  provides  for  competitive  prototypes  before  Milestone  B  approval — 
which  authorizes  entry  into  system  development — unless  the  Milestone  Decision  Authority 
waives  the  requirement.1  Competitive  prototyping,  which  involves  commercial,  government,  or 
academic  sources  producing  early  prototypes  of  weapon  systems  or  critical  subsystems,  can 
help  Department  of  Defense  (DOD)  programs  reduce  technical  risk,  refine  requirements, 
validate  designs  and  cost  estimates,  and  evaluate  manufacturing  processes  prior  to  making 
major  commitments  of  resources.  It  can  also  help  reduce  the  time  it  takes  to  field  a  system,  and 
as  a  result,  reduce  its  acquisition  cost.  WSARA  states  that  the  Milestone  Decision  Authority  may 
waive  the  competitive  prototyping  requirement  only  on  the  basis  that  (1)  the  cost  of  producing 
competitive  prototypes  exceeds  the  expected  life-cycle  benefits  (in  constant  dollars)  of 
producing  such  prototypes,  including  the  benefits  of  improved  performance  and  increased 
technological  and  design  maturity  that  may  be  achieved  through  competitive  prototyping;  or  (2) 
but  for  such  a  waiver,  DOD  would  be  unable  to  meet  critical  national  security  objectives. 

WSARA  also  provides  that  whenever  a  Milestone  Decision  Authority  authorizes  a  waiver  of  the 
competitive  prototyping  requirement  on  the  basis  of  what  WSARA  describes  as  “excessive 
cost,”  the  Milestone  Decision  Authority  is  required  to  submit  notification  of  the  waiver,  together 
with  the  rationale,  to  the  Comptroller  General  of  the  United  States  at  the  same  time  it  is 
submitted  to  the  congressional  defense  committees.  WSARA  further  provides  that  no  later  than 
60  days  after  receipt  of  a  notification  of  a  waiver,  we  are  mandated  to  review  the  rationale  for 
the  waiver  and  submit  a  written  assessment  of  that  rationale  to  the  congressional  defense 
committees.2 

On  July  10,  2013,  we  received  notice  from  DOD  that  it  had  waived  the  competitive  prototyping 
requirement  for  the  VXX  Presidential  Helicopter  Replacement  Program.  The  Navy's  VXX 


Vub.  L.  No.  1 11-23,  §  203(a),  as  amended  by  the  Ike  Skelton  National  Defense  Authorization  Act  for  Fiscal  Year 
2011,  Pub.  L.  No.  111-383,  §  813.  DOD  modified  its  guidance  related  to  the  operation  of  its  acquisition  system 
through  Directive-Type  Memorandum  (DTM)  09-027,  “Implementation  of  Weapon  System  Acquisition  Reform  Act  of 
2009  (Dec.  4,  2009,  incorporating  Change  3,  Dec.  9,  2011).  Major  defense  acquisition  programs  are  those  estimated 
by  DOD  to  require  an  eventual  total  expenditure  for  research,  development,  test,  and  evaluation  of  more  than  $365 
million,  or  for  procurement,  including  all  planned  increments  or  spirals,  of  more  than  $2.19  billion  in  fiscal  year  2000 
constant  dollars.  The  Milestone  Decision  Authority  for  major  defense  acquisition  programs  is  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics;  the  head  of  a  DOD  component;  or  if  delegated  the  component 
acquisition  executive. 

2Pub.  L.  No.  111-23  §  203(b)(1)  &  (2). 
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program  is  an  effort  to  replace  the  aging  fleet  of  19  VH-3D  and  VH-60N  helicopters  used  to 
transport  the  President,  Vice  President,  heads  of  state,  and  others.  The  program  is  the  Navy’s 
second  effort  to  replace  these  helicopters.  The  first  effort,  the  VH-71  program,  was  canceled  in 
2009  due  to  significant  cost  growth,  schedule  delays,  and  performance  issues.  Since  then,  the 
Navy  has  reduced  the  helicopter’s  passenger  capacity,  range,  transportability,  operational 
availability,  airspeed,  and  hover  performance  requirements.  These  performance  trade-offs 
allowed  the  Navy  to  adopt  a  streamlined  acquisition  approach  that  involves  integrating  mature 
communications  and  mission  systems  into  an  existing  commercial  or  military  helicopter  that  is 
already  in  production  without  the  need  for  a  technology  development  phase.  The  Navy  expects 
this  approach  to  lower  the  program’s  cost,  shorten  its  schedule,  and  reduce  its  risk. 

In  this  report,  we  assess  DOD’s  rationale  for  waiving  the  competitive  prototyping  requirement  for 
the  VXX  program  and  the  analysis  used  to  support  it.  To  conduct  our  assessment,  we  compared 
the  rationale  in  the  waiver  to  the  WSARA  requirement  to  determine  the  extent  to  which  the 
waiver  is  consistent  with  the  statute.  In  addition,  we  reviewed  the  Navy’s  cost-benefit  analysis, 
which  provides  the  data  and  assumptions  on  which  the  waiver  is  based.  We  also  questioned 
DOD,  Navy,  and  VXX  program  officials  to  clarify  information  in  this  documentation,  as 
necessary.  Further,  we  relied  on  our  April  2013  report  on  the  VXX  program  for  information  on 
the  degree  to  which  the  cost  estimates  used  to  support  the  Navy’s  cost-benefit  analysis 
complied  with  cost  estimating  and  assessment  best  practices.3  We  did  not  assess  the  estimated 
cost  benefits  of  the  various  prototyping  scenarios  in  the  Navy’s  cost-benefit  analysis  against 
these  best  practices  criteria,  but  did  take  other  steps  to  assess  their  reasonableness. 

We  conducted  this  performance  audit  from  July  2013  to  September  2013  in  accordance  with 
generally  accepted  government  auditing  standards.  Those  standards  require  that  we  plan  and 
perform  the  audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence  obtained 
provides  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit  objectives. 


Results  in  Brief 

DOD’s  rationale  for  waiving  WSARA’s  competitive  prototyping  requirement  for  VXX  addresses 
one  of  the  two  bases  provided  in  the  statute;  namely  that  the  cost  of  producing  competitive 
prototypes  exceeds  the  expected  life-cycle  benefits  (in  constant  dollars)  of  producing  the 
prototypes.  The  VXX  program’s  acquisition  strategy  provided  the  primary  justification  for  the 
prototyping  waiver.  According  to  the  waiver,  VXX  requirements  can  be  met  by  integrating  an 
existing,  in-production,  flight-proven  aircraft  with  mature  mission  systems.  The  Navy  in  its  waiver 
request  also  concluded  that  the  integration  activities  planned  for  the  VXX  program  do  not 
require  additional  technology  maturation  or  risk  reduction  beyond  that  already  being 
accomplished  by  the  government  through  its  own  prototyping  of  certain  critical  mission 
subsystems.  Recognizing  that  the  intent  of  competitive  prototyping  is  to  reduce  cost  and  risk, 
DOD  has  taken  other  actions  that  could  arguably  achieve  these  goals.  Specifically,  DOD 
decided  to  reduce  requirements,  use  an  existing  aircraft,  and  mature  critical  subsystems  before 
integrating  them  on  the  aircraft.  In  the  waiver,  DOD  also  found  reasonable  the  Navy’s  cost- 
benefit  analysis,  which  examined  multiple  acquisition  strategies  with  system-  and  subsystem- 
level  prototyping  from  one  or  two  contractors.  In  all,  the  Navy  examined  six  different  acquisition 


3GAO,  Presidential  Helicopter  Acquisition:  Program  Makes  Progress  in  Balancing  Requirements,  Costs,  and 
Schedule,  GAO-13-257  (Washington,  D.C.:  Apr.  9,  2013). 
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strategies  and  concluded  that  requiring  competitive  prototyping  would  delay  fielding  an  initial 
operational  capability  by  16  months  and  increase  development  costs  by  about  $782  million  to 
$3.38  billion  (in  base  year  201 1  dollars),  depending  on  the  type  and  number  of  prototypes.  The 
Navy  also  estimated  that  the  more  costly  system-level  prototyping  strategies  could  achieve  an 
estimated  $542  million  (in  base  year  201 1  dollars)  in  life-cycle  cost  benefits  by  improving  the 
reliability  of  the  aircraft,  which  in  turn  could  reduce  the  number  of  helicopters  required. 
According  to  the  Navy,  the  cost  data  used  in  its  cost-benefit  analysis  were  drawn  from  the  VXX 
analysis  of  alternatives  study  and  related  activities.  In  April  2013,  we  found  that  the  Navy  used 
cost  estimating  procedures  on  the  VXX  analysis  of  alternatives  that  substantially  complied  with 
best  practices. 


Waiver  Rationale  Is  Consistent  with  WSARA  and  Supporting  Analysis  Incorporated 
Lessons  Learned  from  Prior  Waivers 

DOD’s  rationale  for  waiving  WSARA’s  competitive  prototyping  requirement  for  VXX  addressed 
one  of  the  two  bases  provided  for  a  waiver  in  the  statute,  namely  that  the  cost  of  producing 
competitive  prototypes  exceeds  the  expected  life-cycle  benefits,  including  the  benefits  of 
improved  performance  and  increased  technological  and  design  maturity  that  may  be  achieved 
through  competitive  prototyping.  In  the  waiver,  DOD  also  stated  that  the  cost  of  producing  a 
single  prototype  of  the  VXX  or  its  critical  subsystems  before  Milestone  B  exceeded  the  expected 
life-cycle  benefits.4  The  VXX  program’s  acquisition  strategy  provided  the  primary  justification  for 
the  prototyping  waiver.  According  to  the  waiver,  VXX  requirements  can  be  met  by  integrating  an 
existing,  in-production,  flight-proven  aircraft  with  mature  mission  systems.  The  program  is 
conducting  government-led  prototyping  of  certain  critical  mission  subsystems,  including  an 
Internet  Protocol-based  communication  capability  that  is  comprised  of  existing  commercial 
equipment.  We  have  previously  found  that  there  are  potential  risks  associated  with  integrating 
government-developed  mission  systems  into  a  new  airframe.5  The  Navy’s  waiver  request 
acknowledged  that  integration  could  be  a  challenge,  but  it  concluded  that  the  integration 
activities  planned  for  the  VXX  program  do  not  require  technology  maturation  or  risk  reduction 
beyond  that  already  being  accomplished  by  the  government.  Therefore,  it  concluded  that  the 
program  would  not  benefit  from  additional  prototyping  at  the  system  or  subsystem  level. 
Recognizing  that  the  intent  of  competitive  prototyping  is  to  reduce  cost  and  risk,  DOD  has  taken 
other  actions  that  could  arguably  achieve  these  goals.  Specifically,  DOD  decided  to  reduce 
requirements,  use  an  existing  aircraft,  and  mature  critical  subsystems  before  integrating  them 
on  the  aircraft. 

In  the  waiver,  DOD  found  reasonable  the  Navy’s  cost-benefit  analysis,  which  examined  multiple 
acquisition  strategies  with  system-  and  subsystem-level  prototyping  from  one  or  two  contractors. 
The  Navy’s  approach  was  consistent  with  a  key  principle  in  DOD’s  policy  on  economic  analysis, 
which  states  that  each  feasible  alternative  for  meeting  an  objective  must  be  considered,  and  its 
life-cycle  costs  and  benefits  evaluated.6  In  all,  the  Navy  examined  six  different  acquisition 


4WSARA  provides  that  whenever  a  Milestone  Decision  Authority  authorizes  a  competitive  prototyping  waiver,  the 
program  is  required  to  produce  a  prototype  prior  to  Milestone  B  approval  if  the  expected  life-cycle  benefits  (in 
constant  dollars)  of  producing  such  prototype  exceed  its  cost  and  its  production  is  consistent  with  achieving  critical 
national  security  objectives.  Pub.  L.  No.  1 11-23  §  203(a)(3)(A). 

5GAO-1 3-257. 

6DOD  Instruction  7041.3,  Economic  Analysis  for  Decisionmaking  ( Nov.  7,  1995). 
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strategies  with  various  degrees  of  prototyping  and  compared  the  development  cost  of  the  VXX 
as  well  as  the  potential  life-cycle  benefits  of  prototyping  under  each  scenario.  Based  on  this 
analysis,  DOD  concluded  that  requiring  competitive  prototyping  of  critical  subsystems  would 
delay  fielding  an  initial  operational  capability  by  16  months  and  increase  development  costs  by 
about  $1,189  billion  for  two  industry  prototypes  and  $782  million  for  a  single  prototype  in  base 
year  201 1  dollars.  The  waiver  also  concluded  that  pursuing  an  aircraft  prototyping  strategy  to 
improve  aircraft  performance  would  increase  development  costs  by  about  $3.38  billion  for  a 
strategy  with  two  industry  prototypes  and  $2.84  billion  for  a  single  prototype  in  base  year  201 1 
dollars.  Table  1  includes  a  description  of  each  of  these  acquisition  strategies  and  their  total 
estimated  development  cost,  schedule,  and  expected  benefits. 


Table  1:  Navy’s  Estimated  Cost,  Schedule,  and  Benefits  for  Various  VXX  Acquisition  Strategies 

Acquisition  strategy 

Total  development  cost 
including  prototyping  (in 
base  year  2011  billions) 

Time  to  field  initial 
capability  (in  months) 

Expected  benefits  of 
prototyping 

Planned  strategy  with 
government  prototyping  of 
critical  mission  subsystems 
and  no  aircraft  prototyping3 

$4.33-4.51 

85 

Subsystem  prototypes 

One  contractor  prototyping 
critical  mission  subsystems 

5.29 

101 

None 

Two  contractors  prototyping 
critical  mission  subsystems 

5.70 

101 

None 

Aircraft  prototypes D 

Major  development  program 
with  one  aircraft  prototype 

7.17 

At  least  101 

Life  cycle  cost  savings  of 
$542  million  (base  year 

201 1  dollars)  driven  by 
improved  operational 
availability  and  reduction  in 
number  of  helicopters 
required;  potential  capability 
improvements 

Major  development  program 
with  two  competing  aircraft 
prototypes 

7.71 

At  least  101 

Life  cycle  cost  savings  of 
$542  million  (base  year 

201 1  dollars)  driven  by 
improved  operational 
availability  and  reduction  in 
number  of  helicopters 
required;  potential  capability 
improvements 

Source:  GAO  analysis  of  Navy’s  VXX  cost-benefit  analysis. 

aThis  strategy  was  treated  as  two  separate  scenarios  in  the  Navy’s  cost-benefit  analysis. 
bAssumes  government  prototyping  of  critical  mission  subsystems. 


The  Navy  used  cost  estimating  procedures  on  the  VXX  competitive  prototyping  cost-benefit 
analysis  that  were  substantially  compliant  with  best  practices.7  According  to  the  Navy,  the  cost 
data  used  in  its  cost-benefit  analysis  were  drawn  from  the  VXX  analysis  of  alternatives  study 
and  related  activities.  This  analysis  was  completed  from  2010  to  2012  and  provided  the  basis 
for  the  program’s  current  acquisition  strategy.  In  our  April  2013  report,  we  found  that  the  cost 


7GAO,  GAO  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and  Managing  Capital  Program 
Costs,  GAO-09-3SP  (Washington,  D.C.:  Mar.  2,  2009). 
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estimating  procedure  used  for  the  analysis  of  alternatives  was  substantially  compliant  with  best 
practices  criteria.8  Specifically,  we  found  the  analysis  of  alternatives  cost  estimate  to  be 
comprehensive  and  well  documented.  We  also  found  that  it  was  substantially  accurate  and 
partially  met  our  criteria  for  being  credible.  The  cost  estimating  best  practices  associated  with 
each  of  those  characteristics  used  in  assessing  the  analysis  of  alternatives  are  provided  in 
enclosure  1 . 

The  Navy  incorporated  lessons  learned  from  DOD’s  first  two  competitive  waivers  to  improve 
certain  aspects  of  the  VXX  cost-benefit  analysis.  In  an  August  2012  report,  we  concluded  that 
the  rationale  for  the  Enhanced  Polar  System  program  waiver  was  consistent  with  WSARA,  but 
the  cost-benefit  analysis  supporting  it  was  incomplete.9  That  analysis  did  not  include  a  specific 
dollar  value  or  range  of  values  for  the  estimated  life-cycle  benefits  of  competitive  prototyping. 
We  also  found  that  the  Air  Force’s  estimate  of  the  cost  of  competitive  prototyping  was  prepared 
by  the  program  office  and  not  independently  estimated  or  reviewed  by  other  DOD  cost 
estimating  organizations.  In  a  March  2013  report,  we  found  that  the  Air  Force  only  evaluated 
one  potential  approach  to  implementing  competitive  prototyping  in  its  waiver  request  for  the 
Combat  Rescue  Helicopter  program.10  We  concluded  that  the  Air  Force  could  have  more  fully 
evaluated  the  cost  and  benefits  of  additional  prototyping  strategies,  including  more  limited 
approaches  that  are  mentioned  in  WSARA  or  DOD’s  implementing  memorandum,  such  as 
producing  a  single  prototype  or  prototyping  critical  subsystems  before  Milestone  B.  The  VXX 
cost-benefit  analysis  addressed  all  of  these  points. 


8GAO-1 3-257. 

9GAO,  Department  of  Defense’s  Waiver  of  Competitive  Prototyping  Requirements  for  Enhanced  Polar  System 
Program,  GAO-12-983R  (Washington,  D.C.:  Aug.  23,  2012). 

10GAO,  Department  of  Defense’s  Waiver  of  Competitive  Prototyping  Requirements  for  Combat  Rescue  Helicopter 
Program,  GAO-1 3-31 3R  (Washington,  D.C.:  Mar.  7,  2013). 
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Agency  Comments  and  Our  Evaluation 

We  provided  a  draft  of  this  report  to  DOD  for  comment.  In  its  written  comments,  which  are 
reprinted  in  enclosure  2,  DOD  concurred  with  our  assessment  of  the  VXX  prototyping  waiver. 


We  are  sending  copies  of  this  report  to  interested  congressional  committees,  the  Secretary  of 
Defense,  the  Secretary  of  the  Navy,  and  the  Commandant  of  the  Marine  Corps.  In  addition,  the 
report  will  be  available  at  no  charge  on  the  GAO  website  at  http://www.gao.gov. 

If  you  or  your  staff  have  any  questions,  please  contact  me  at  (202)  51 2-4841  or 
sullivanm@gao.gov.  Contact  points  for  our  Offices  of  Congressional  Relations  and  Public  Affairs 
may  be  found  on  the  last  page  of  this  report.  GAO  staff  who  made  key  contributions  to  this 
report  were  Ron  Schwenn,  Assistant  Director;  Bruce  Thomas,  Assistant  Director;  Marie  P. 
Ahearn;  Pedro  Almoguera;  Jerry  Clark;  Bonita  Oden;  Kenneth  E.  Patton;  and  Carol  Petersen. 


Michael  J.  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
Enclosures  -  2 
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List  of  Committees 


The  Honorable  Carl  Levin 
Chairman 

The  Honorable  James  Inhofe 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Richard  J.  Durbin 
Chairman 

The  Honorable  Thad  Cochran 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Howard  P.  “Buck”  McKeon 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  C.W.  Bill  Young 
Chairman 

The  Honorable  Pete  Visclosky 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Enclosure-1 


Cost  Estimating  Best  Practices 


Characteristic 

Best  practices 

Comprehensive 

The  cost  estimate  includes  all  life  cycle  costs. 

The  cost  estimate  completely  defines  the  program,  reflects  the  current  schedule,  and 
is  technically  reasonable. 

The  cost  estimate  work  breakdown  structure  (WBS)  is  product-oriented,  traceable  to 
the  statement  of  work/objective,  and  at  an  appropriate  level  of  detail  to  ensure  that 
cost  elements  are  neither  omitted  nor  double-counted. 

The  estimate  documents  all  cost-influencing  ground  rules  and  assumptions. 

Well  documented 

The  documentation  should  capture  the  source  data  used,  the  reliability  of  the  data, 
and  how  the  data  were  normalized. 

The  documentation  describes  in  sufficient  detail  the  calculations  performed  and  the 
estimating  methodology  used  to  derive  each  element’s  cost. 

The  documentation  describes  step  by  step  how  the  estimate  was  developed  so  that  a 
cost  analyst  unfamiliar  with  the  program  could  understand  what  was  done  and 
replicate  it. 

The  documentation  discusses  the  technical  baseline  description  and  the  data  in  the 
baseline  is  consistent  with  the  estimate. 

The  documentation  provides  evidence  that  the  cost  estimate  was  reviewed  and 
accepted  by  management. 

Accurate 

The  cost  estimate  results  are  unbiased,  not  overly  conservative  or  optimistic  and 
based  on  an  assessment  of  most  likely  costs. 

The  estimate  has  been  adjusted  properly  for  inflation. 

The  estimate  contains  few,  if  any,  minor  mistakes. 

The  cost  estimate  is  regularly  updated  to  reflect  significant  changes  in  the  program  so 
that  it  is  always  reflecting  current  status.  Variances  between  planned  and  actual  costs 
are  documented,  explained,  and  reviewed. 

The  estimate  is  based  on  a  historical  record  of  cost  estimating  and  actual  experiences 
from  other  comparable  programs. 

Credible 

The  cost  estimate  includes  a  sensitivity  analysis  that  identifies  a  range  of  possible 
costs  based  on  varying  major  assumptions,  parameters,  and  data  inputs. 

A  risk  and  uncertainty  analysis  was  conducted  that  quantified  the  imperfectly 
understood  risks  and  identified  the  effects  of  changing  key  cost  driver  assumptions 
and  factors. 

Major  cost  elements  were  cross-checked  to  see  whether  results  were  similar. 

An  independent  cost  estimate  was  conducted  by  a  group  outside  the  acquiring 
organization  to  determine  whether  other  estimating  methods  produce  similar  results. 

Source:  GAO. 
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Enclosure-2 


Comments  from  the  Department  of  Defense 


ASSISTANT  SECRETARY  OF  DEFENSE 

30 1  5  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3015 


AUG  30  2013 


Mr.  Michael  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  NW 
Washington,  DC  20548 

Dear  Mr.  Sullivan: 

This  is  the  Department  of  Defense  response  to  the  Government  Accountability  Office  (GAO) 
Draft  Report,  GAO-1 3-862R,  “COMPETITIVE  PROTOTYPING:  Department  of  Defense’s  Waiver 
of  the  Competitive  Prototyping  Requirement  for  the  VXX  Presidential  Helicopter  Replacement 
Program’’  dated  August  22,  2013  (GAO  Code  121 169).  The  Department  appreciates  the  effort  of  the 
GAO  and  the  opportunity  to  comment  on  the  draft  report. 

The  Department  concurs  with  and  appreciates  GAO’s  conclusions  that  the  VXX  waiver 
rationale  is  consistent  with  Weapons  Systems  Acquisition  Reform  Act  of  2009,  Public  Law  111- 
23  and  that  supporting  analysis  incorporated  lessons  learned  from  prior  waivers.  We  have  no 
additional  comments  on  the  draft  report. 


(121169) 
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